1. Field of the Invention
The present invention relates to an image processing method, an image processing device, a recording medium, and a transmission medium. In particular, the present invention relates to an image processing method for correcting a pupil region in an image which corresponds to the pupil of a person and in which unsatisfactory color tone arises in the image, to an image processing device to which the image processing method is applied, to a recording medium on which a program is recorded for realizing the image processing method on a computer, and to a transmission medium for transmitting the program for realizing the image processing method by a computer.
2. Description of the Related Art
When a subject is photographed by a camera, the color tone of the subject in the photographed and recorded image may greatly differ from the actual color tone due to the reflected state of the light at the time of photographing. For example, when a person who is the subject is photographed from the front by using a flash, in a dark place, the flash light is, from the front, incident on and regularly reflects off of the eye portion of a person who has their pupils open. When this state is photographed and recorded as an image, the region (pupil region) corresponding to the pupil on the image may take on a red color (so-called xe2x80x9cred-eyexe2x80x9d) or may take on a gold color (so-called xe2x80x9cgold-eyexe2x80x9d). Such an unsatisfactory color tone in the pupil region of an image has an extremely unnatural and unpleasant appearance. Thus, conventionally, various methods have been proposed for correcting the deficient color tone in the pupil region by image processing such that the pupil region will appear natural.
For example, Japanese Patent Application Laid-Open (JP-A) No. 7-72537 discloses the following technique. A region in which there is the possibility that unsatisfactory color tone of a pupil exists is determined. By comparing the hue or the brightness or the like of sampled pixels within this region with a threshold value, pixels which have the possibly of being an unsatisfactory color tone of a pupil are determined, and a region which corresponds to the main portion of a pupil having an unsatisfactory tone is determined. The RGB values of the respective pixels within this region are converted to YCC values, and the values of the chromaticity channels C1, C2 are set to zero so that the chromaticity is eliminated. A predetermined value (e.g., 0.35) which is less than one and which has been determined by experimentation is multiplied by the value of the brightness channel Y so as to lower the brightness. In this way, the unsatisfactory color tone is corrected.
Further, JP-A-9-261580 discloses the following technique. A pupil candidate region (a region which appears to be a pupil) is selected on the basis of color information within a region surrounded by an edge. Pixels having unsatisfactory color tone (red-eye) are extracted from the selected pupil candidate region. For each of the pixels, for example, the R, G, B values are made to coincide with the lowest of the R, G, B values, or the R value is made to coincide with the average value of G and B, or on the basis of color information of a color selected from a color sample by an operator, the R, G, B values are changed, so as to correct the unsatisfactory color tone.
However, the characteristic amounts (e.g., brightness, saturation, hue or the like) of a pupil region in a case in which there is no unsatisfactory color tone in the pupil region in the image, greatly differ in accordance with the photographing conditions, such as the lighting conditions, at the time of photographing. Thus, how the pupil region in an image in which unsatisfactory color tone arises should be corrected differs from image to image. Further, how the characteristic amounts of the pupil region vary from the appropriate values due to the unsatisfactory color tone also differs in accordance with the photographing conditions, such as, for example, the distance to the photographed subject or the like. Thus, it is difficult to estimate the appropriate values from the characteristic amounts of pupil regions in which unsatisfactory color tone arises.
In the technique disclosed in JP-A-7-72537, for a region which is judged to have an unsatisfactory color tone, the chromaticity is eliminated for each pixel, and correction is carried out uniformly by multiplying the brightness by a constant coefficient. Further, in the technique disclosed in JP-A-9-261580, correction is carried out only by using the characteristic amounts (specifically, the R, G, B values of each pixel) of a region judged to have unsatisfactory color tone. Thus, an image whose unsatisfactory color tone regions have been corrected often has an unnatural feel.
Further, there are many cases in which unsatisfactory color tone of the pupil regions arises in a pair of pupil regions corresponding to the pupils of both eyes of the same person. However, there are cases in which the degree of unsatisfactory color tone arising in the pair of pupil regions differs (for each pupil region) due to the angle at which the light of the flash is illuminated onto the person who is the subject, or the like. The technologies disclosed in the aforementioned publications do not take into consideration such correction of unsatisfactory color tone in a pair of pupil regions, and have the problem that a pair of pupil regions (and the overall image) are often finished with an unnatural feel due to the respective finishes of the pair of pupil regions after unsatisfactory color tone correction has been carried out greatly differing (for each pupil region).
In view of the aforementioned, a first object of the present invention is to provide an image processing method, an image processing device, a recording medium, and a transmission medium which enable correction of unsatisfactory color tone in pupil regions in an image so that the image will be finished with a natural feel.
A second object of the present invention is to provide an image processing method, an image processing device, a recording medium and a transmission medium which enable correction of unsatisfactory color tone within pupil regions such that a pair of pupil regions in an image, which correspond to the pupils of a single person, are finished with a natural feel.
As discussed above, characteristic amounts, such as brightness, saturation, hue, and the like of pupil regions in an image in cases in which unsatisfactory color tone does not arise therein (namely, desirable values of characteristic amounts of pupil regions for finishing the image with a natural feel) greatly differ in accordance with the photographing conditions such as the illumination conditions at the time of photographing. The characteristic amounts of a region including the majority portion of the image and the characteristic amounts of the region in the vicinity of the pupil regions within the image greatly differ in accordance with the photographing conditions regardless of whether or not unsatisfactory color tone has arisen in the pupil regions in the image. On the basis of these facts, the present inventors arrived at the present invention by hypothesizing that, for preferable values of characteristic amounts of pupil regions for finishing an image with a natural feel, the characteristic amounts of regions including the majority of the image and the characteristic amounts of the region in a vicinity of the pupil region are strongly correlated, and preferable values of the characteristic amounts of the pupil regions can be estimated from the characteristic amounts of the regions including the majority of the image and the characteristic amounts of the region in a vicinity of the pupil region.
On the basis of the above, in order to achieve the first object, an image processing method comprising the steps of: extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; determining a target value of a characteristic amount of the pupil region in which unsatisfactory color tone has arisen, on the basis of a characteristic amount of a region including a majority of the image, a characteristic amount of a region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, and a characteristic amount of the pupil region in which unsatisfactory color tone has arisen; and correcting a value of each pixel within the pupil region in which unsatisfactory color tone has arisen, such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is substantially equal to the target value as first aspect of the invention is provided.
In the invention of the first aspect, a pupil region, which corresponds to a pupil of a person and in which unsatisfactory color tone (e.g., red-eye, gold-eye, or the like) has arisen, is extracted from an image. The extraction of the pupil region in which unsatisfactory color tone has arisen may be carried out as follows for example. For each pixel of an image region which includes a pupil region in which unsatisfactory color tone has arisen (the region may be designated by an operator, or may be extracted automatically by image analysis), at least one characteristic amount of among brightness, saturation and hue is determined. A three-dimensional xyz space is set with the image region as the xy plane and the characteristic amount as the z axis. The xy plane is divided per region in which the z axis values (characteristic amounts) are distributed in a mountain-shape with respect to the spreading of the xy plane. On the basis of at least one of the configuration, position, surface area, and statistical characteristic amount on the xy plane of each divisional region, a pupil region which corresponds to the pupil of a person and in which unsatisfactory color tone has arisen is determined.
In the invention of the first aspect, the target value of the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is determined on the basis of the characteristic amount of a region including the majority of the image, the characteristic amount of a region in the vicinity of the pupil region in which unsatisfactory color tone has arisen, and the characteristic amount of the pupil region in which unsatisfactory color tone has arisen. The region which includes the majority of the image (also referred to as xe2x80x9cmajority regionxe2x80x9d) may be, for example, the entire image, or may be a region which remains after removing from the image the peripheral edge portion of the image or a region which is separated, by a fixed distance or more on the image, from the pupil region in which unsatisfactory color tone has arisen, or may be the majority of a trimmed region in the case that the region including the pupil region in which unsatisfactory color tone has arisen is trimmed from the image. Further, the region in the vicinity of the pupil region may be an arbitrary shape such as circular, oval, rectangular, a configuration resembling a pupil region, or the like. The size of the region in the vicinity of the pupil region may be varied in accordance with the surface area of the pupil region, or the size thereof may be fixed.
Any of various known characteristic amounts can be used as the characteristic amount. However, as in the second aspect of the present invention, at least one of the average value of brightness, the average value of saturation, the average value of hue, and a value corresponding to the minimum value of brightness (e.g., the minimum value of brightness or the brightness of a shadow portion), may be used as the characteristic amount of the region including the majority of the image, the characteristic amount of the region in the vicinity of the pupil region in which unsatisfactory color tone has arisen, and the characteristic amount of the pupil region in which unsatisfactory color tone has arisen. As the target value of the characteristic amount of the pupil region in which unsatisfactory color tone has arisen, a target value of an average value of brightness, a target value of an average value of saturation, and a target value of an average value of hue may be used.
As described above, the characteristic amount of a region including the majority of the image and the characteristic amount of the region in a vicinity of the pupil region in which unsatisfactory color tone has arisen are strongly correlated with a desirable value of the characteristics amount of the pupil region for finishing the image with a natural feel. Thus, as described above, by determining the target value of the characteristic amount of the pupil region by using the characteristic amount of the region including the majority of the image or the characteristic amount of the region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, the target value of the characteristic amount of the pupil region can be determined such that the image can be finished with a natural feel.
In the invention of the first aspect, the value of each pixel within the pupil region in which unsatisfactory color tone has arisen is corrected such that the characteristic amount of a pupil region in which unsatisfactory color tone has arisen substantially coincides with the target value. Thus, the unsatisfactory color tone of the pupil region in the image can be corrected such that the image is finished with a natural feel. The correction of the value of each pixel within the pupil region such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen substantially coincides with the determined target value, is desirably carried out as follows for example: a conversion formula is determined which converts the value of each pixel such that the average value of the characteristic amounts of the pupil region in which unsatisfactory color tone has arisen substantially coincides with the determined target value. By converting the value of each pixel in accordance with this conversion formula, the value of each pixel can be corrected such that even the dispersion in the values of the respective pixels within the pupil region can be reflected.
In a case in which, as in the second aspect of the present invention, an average value of brightness and a value corresponding to the lowest value of brightness of the region including the majority of the image, are included in the characteristic amount of the region including the majority of the image, and the average value of the brightness of the region in the vicinity of the pupil region in which unsatisfactory color tone has arisen is included in the characteristic amount of the region in the vicinity of the pupil region in which unsatisfactory color tone has arisen, and the target value of the average value of brightness of the pupil region is included in the target value of the characteristic amount of the pupil region, it is preferable that, as in the third aspect of the present invention, the target value of the average value of the brightness of the pupil region is determined so as to be less than or equal to the average value of brightness of the region including the majority of the image, and so as to be less than or equal to a value corresponding to the minimum value of the brightness of the region in the vicinity of the pupil region, and so as to be greater than or equal to a value corresponding to the minimum value of the brightness in the region including the majority of the image.
Generally, the pupil region has a low brightness. Thus, by making the target value of the average value of brightness of the pupil region in which unsatisfactory color tone has arisen to be less than or equal to an average value of brightness of the region including the majority of the image, the target value can be made to be close to the desired brightness of the image. Generally, the brightness of the pupil region is less than the brightness of the eyelashes, the eyebrows or the iris at the periphery of the pupil. Thus, by making the target value of the average value of the brightness of the pupil region to be less than or equal to a value corresponding to the minimum value of the brightness of the region in a vicinity of the pupil region, the target value can be made to be close to the desirable brightness of the image.
Usually, an image which has been photographed by using a flash has low contrast. In such an image, if the brightness of the pupil region is too low, it will appear as if the brightness has been artificially lowered. Thus, by setting the target value of the average value of the brightness of the pupil region to be greater than or equal to a value corresponding to a minimum value of brightness of the region including the majority of the image, the target value can be prevented from becoming a value which is inappropriately low, and the target value can be made to be close to a desirable brightness of the image.
In this way, in accordance with the third aspect of the present invention, the target value of the average value of the brightness of the pupil region in which unsatisfactory color tone has arisen is determined such that the aforementioned respective conditions are satisfied. Thus, the target value of the average value of the brightness of the pupil region can be set to a value which is suitable for the image to be finished with a natural feel.
In the second aspect of the present invention, in a case in which the average value of the saturation of the region in a vicinity of the pupil region in which unsatisfactory color tone has arisen is included in the characteristic amount of the region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, and the target value of the average value of the saturation of the pupil region is included in the average value of the characteristic amount of the pupil region, as in the fourth aspect of the present invention, it is preferable that the target value of the average value of the saturation of the pupil region is determined so as to be less than or equal to the average value of the saturation of the region in a vicinity of the pupil region and such that the achromaticity is not zero.
Generally, flesh colored regions take up a large surface area within the region in the vicinity of the pupil region, and the region in the vicinity of the pupil region includes regions corresponding to objects which are nearly achromatic (i.e., whose saturation is zero) such as eyelashes and eyebrows. Thus, the average value of the saturation of the region in the vicinity of the pupil region is a saturation value which is classified as being low saturation in the entire image. In the fourth aspect of the present invention, the target value of the average value of the saturation of the pupil region in which unsatisfactory color tone occurs is determined so as to be less than or equal to the average value of the saturation of the region in the vicinity of the pupil region and such that the saturation is not zero. Thus, the target value of the average value of the saturation of the pupil region can be made to be a value which is appropriate for the image to be finished with a natural feel.
In the second aspect of the present invention, in a case in which the average value of the hue of the region in the vicinity of the pupil region in which unsatisfactory color tone has arisen is included in the characteristic amount of the region in the vicinity of the pupil region in which unsatisfactory color tone has arisen, and the target value of the average value of the hue of the pupil region is included in the target value of the characteristic amount of the pupil region, as in the fifth aspect, the target value of the average value of the hue of the pupil region is preferably determined so as to be substantially equal to the average value of the hue of the region in the vicinity of the pupil region.
The inventors of the present application analyzed the results of functional evaluation on images in which pupil regions which had had unsatisfactory color tones were corrected, and found that the results of evaluation by functional evaluation of the hue of the pupil region after unsatisfactory color tone correction were better for hues corresponding to warm colors than for hues corresponding to achromatic colors. As described above, because flesh color regions take up a large surface area in the regions in a vicinity of the pupil region, the average value of the hue of the region in the vicinity of the pupil region is a value corresponding to a warm color. In the fifth aspect of the present invention, the target value of the average value of the hue of the pupil region in which unsatisfactory color tone has arisen is determined so as to be substantially equal to the average value of the hue of the region in a vicinity of the pupil region. Thus, the target value of the average value of the hue of the pupil region can be made to be an appropriate value having good functional evaluation results and resulting in an image with a natural feel.
In order to achieve the second object, a sixth aspect of the present invention is an image processing method comprising the steps of: extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; in a case in which a pair of pupil regions corresponding to a pair of pupils of the person exist in the image and unsatisfactory color tone has arisen in at least one pupil region of the pair of pupil regions, determining a target value of a characteristic amount of the pupil region in which unsatisfactory color tone has arisen such that characteristic amounts of the pair of pupil regions become substantially a same value; and correcting a value of each pixel in the pupil region in which unsatisfactory color tone has arisen, such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is substantially equal to the determined target value.
In the sixth aspect of the present invention, when unsatisfactory color tone has arisen in at least one pupil region of a pair of pupil regions corresponding to a pair of pupils of the same person, the target value of the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is determined such that the characteristic amounts of the pair of pupil regions become substantially the same values. Thus, after the unsatisfactory color tone of the pupil region in which unsatisfactory color tone has arisen has been corrected, the characteristic amounts of the pupil regions of the pair of pupil regions can be prevented from differing greatly.
In the sixth aspect of the present invention, the value of each pixel in the pupil region in which unsatisfactory color tone has arisen is corrected such that the characteristic amount of the region in which unsatisfactory color tone has arisen substantially coincides with the determined target value. Thus, the finishes of the respective pupil regions of the pair of pupil regions whose unsatisfactory color tones have been corrected are substantially the same, and the unsatisfactory color tones of the pupil regions can be corrected such that the pair of pupil regions in the image corresponding to a pair of pupils of the same person can be finished with a natural feel.
In the sixth aspect of the present invention, in a case in which unsatisfactory color tone arises in only one pupil region of a pair of pupil regions, the target value of the characteristic amount for the one pupil region may be determined, for example, on the basis of only the characteristic amount of the other pupil region in which no unsatisfactory color tone has arisen. As in the first aspect of the present invention, the target value may be determined by also taking into account the characteristic amount of the region including the majority of the image, the characteristic amount of the region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, and the characteristic amount of the pupil region in which unsatisfactory color tone has arisen.
Moreover, in the sixth aspect of the present invention, in a case in which unsatisfactory color tone arises in both pupil regions of a pair of pupil regions, the target values of the characteristic amounts of the pupil regions may, for example, be determined by, after the target value of the characteristic amount of each pupil region is determined as in the first aspect, effecting correction such that the respective target values become substantially the same. Or, a single target value to be used for both pupil regions may be determined on the basis of the characteristic amount of the region including the majority of the image, the characteristic amounts of regions in vicinities of both pupil regions, and the characteristic amounts of both pupil regions. A seventh aspect of the present invention is an image processing device comprising: an extracting device for extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; a first determining device for determining a target value of a characteristic amount of the pupil region in which unsatisfactory color tone has arisen, on the basis of a characteristic amount of a region including a majority of the image, a characteristic amount of a region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, and a characteristic amount of the pupil region in which unsatisfactory color tone has arisen; and a correcting device for correcting a value of each pixel in the pupil region in which unsatisfactory color tone has arisen, such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is substantially equal to the target value.
Thus, in the same way as in the first aspect, the unsatisfactory color tone of the pupil region in the image can be corrected such that the image is finished with a natural feel. An eighth aspect of the present invention is an image processing device comprising: an extracting device for extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; a second determining device for, in a case in which a pair of pupil regions corresponding to a pair of pupils of the person exist in the image and unsatisfactory color tone has arisen in at least one pupil region of the pair of pupil regions, determining a target value of a characteristic amount of the pupil region in which unsatisfactory color tone has arisen such that characteristic amounts of the pair of pupil regions become substantially a same value; and a correcting device for correcting a value of each pixel in the pupil region in which unsatisfactory color tone has arisen, such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is substantially equal to the target value.
Thus, in the same way as in the sixth aspect, the unsatisfactory color tones of the pupil regions can be corrected such that the pair of pupil regions in the image corresponding to a pair of pupils of the same person can be finished with a natural feel.
A ninth aspect of the present invention is a recording medium on which is recorded a program for executing, at a computer, a process comprising: a first step of extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; a second step of determining a target value of a characteristic amount of the pupil region in which unsatisfactory color tone has arisen, on the basis of a characteristic amount of a region including a majority of the image, a characteristic amount of a region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, and a characteristic amount of the pupil region in which unsatisfactory color tone has arisen; and a third step of correcting a value of each pixel within the pupil region in which the unsatisfactory color tone has arisen such that the characteristic amount of the pupil region in which the unsatisfactory color tone has arisen is substantially equal to the target value.
Instructions for executing the above-described steps 1 through 3, i.e., a program for realizing the image processing method of the first aspect on a computer, are recorded on the recording medium of the ninth aspect of the present invention. Thus, by a computer reading and executing the program recorded on the recording medium, in the same way as in the first aspect of the present invention, the unsatisfactory color tone of a pupil region in an image can be corrected such that the image is finished with a natural feel. A tenth aspect of the present invention is a recording medium on which is recorded a program for executing, at a computer, a process comprising: a first step of extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen is substantially equal to the target value.
Instructions for executing the above-described steps 1 through 3, i.e., a program for realizing the image processing method of the sixth aspect on a computer, are recorded on the recording medium of the tenth aspect of the present invention. Thus, by a computer reading and executing the program recorded on the recording medium, in the same way as in the sixth aspect of the present invention, the unsatisfactory color tones of a pair of pupil regions corresponding to the pair of pupils of the same person can be corrected such that the image is finished with a natural feel.
An eleventh aspect of the present invention is a transmission medium for transmitting a program for executing, at a computer, a process comprising: a first step of extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; a second step of determining a target value of a characteristic amount of the pupil region in which unsatisfactory color tone has arisen, on the basis of a characteristic amount of a region including a majority of the image, a characteristic amount of a region in a vicinity of the pupil region in which unsatisfactory color tone has arisen, and a characteristic amount of the pupil region in which unsatisfactory color tone has arisen; and a third step of correcting a value of each pixel within the pupil region in which unsatisfactory color tone has arisen, such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is substantially equal to the target value.
The transmission medium of the eleventh aspect of the present invention transmits a program for realizing, by a computer, a process including the above-described steps 1 through 3, i.e., for realizing the image processing method of the first aspect. Thus, by a computer temporarily storing in a storing means a program which has been transmitted by the transmission medium, and thereafter, the computer reading the program from the storing means and executing the program, in the same way as in the first aspect, the unsatisfactory color tone of a pupil region in an image can be corrected such that the image is finished with a natural feel.
A twelfth aspect of the present invention is a transmission medium for transmitting a program for executing, at a computer, a process comprising: a first step of extracting, from an image, a pupil region which corresponds to a pupil of a person and in which unsatisfactory color tone has arisen on the image; a second step of, in a case in which a pair of pupil regions corresponding to a pair of pupils of the person exist in the image and unsatisfactory color tone has arisen in at least one pupil region of the pair of pupil regions, determining a target value of a characteristic amount for the pupil region in which the unsatisfactory color tone has arisen such that characteristic amounts of the pair of pupil regions become substantially a same value; and a third step of correcting a value of each pixel in the pupil region in which unsatisfactory color tone has arisen, such that the characteristic amount of the pupil region in which unsatisfactory color tone has arisen is substantially equal to the determined target value.
The transmission medium of the twelfth aspect of the present invention transmits a program for realizing, by a computer, a process including the above-described steps 1-3, i.e., for realizing the image processing method of the sixth aspect. Thus, by a computer temporarily storing in a storing means a program which has been transmitted by the transmission medium, and thereafter, the computer reading the program from the storing means and executing the program, in the same way as in the sixth aspect, the unsatisfactory color tones of a pair of pupil regions corresponding to the pupils of the same person can be corrected such that the image is finished with a natural feel.